Recently, a method of recognizing an object using machine learning techniques has been studied. As one of these machine learning techniques, a deep learning technique, which uses a neural network with several hidden layers between its input layer and its output layer, has a high recognizing performance.
And, the neural network using the deep learning technique is generally learned through backpropagation using one or more losses.
However, upon learning by backpropagation, as a learning rate, i.e., a ratio of reflecting the losses on one or more parameters of the neural network, gets smaller, the learning becomes more sophisticated, but it takes a long time in learning the neural network.
Hence, according to the conventional art, the neural network is learned in such a manner that reduces the learning rate by a certain amount after a certain number of iterative learning. That is, in an initial stage of the learning, the learning rate is set as high and thus the fluctuation range of the parameters may be increased, resulting in a fast learning. Thereafter, the learning rate is decreased stepwise and thus the fluctuation range of the parameters may be reduced, resulting in allowing the parameters to be adjusted minutely.
A method of learning the neural network according to the conventional art will be described as follows by referring to FIGS. 1A and 1B.
First of all, initial constant values for the learning rate of the neural network are set.
Herein, the initial constant values may include (i) a base learning rate (lr) which is an initial learning rate, (ii) a step which is an iteration unit for changing the learning rate, and (iii) a gamma which is a value for adjusting a change rate of the learning rate. Herein, the iteration may be the number of repetitions of the learning by using one batch data.
Then, the learning rate can be expressed as the following equation 1.
                    current_lr        =                  base_lr          ×                      gamma                          (                                                current                  ⁢                                                                          ⁢                  _                  ⁢                                                                          ⁢                  iter                                step                            )                                                          [                  Equation          ⁢                                          ⁢          1                ]            
FIG. 1A shows a state in which the learning rate is changed when the base learning rate (lr) is set to 0.01, the step is set to 100,000, and the gamma is set to 0.1, and FIG. 1B shows a state in which a loss is changed when the neural network is learned by using the changed learning rate in FIG. 1A.
Referring to FIG. 1A, the neural network is learned through backpropagation by referring to each loss for each iteration by using the learning rate of 0.01 as the base learning rate, from the beginning to 100,000-th iterations. Then, when the accumulated number of iterations reaches 100,000, the learning rate is changed to 0.001 according to Equation 1, and the learning is performed until 200,000-th iteration. Also, when the accumulated number of iterations reaches 200,000, the learning rate is changed to 0.0001 according to Equation 1, and the learning is performed until 300,000-th iteration. Then, the learning process is repeated in the same manner.
However, as can be seen from FIG. 1B, the conventional learning method reduces the learning rate at a fixed rate regardless of the loss, and thus experiments should be performed several times in order to find optimal values of the step and the gamma.
Also, even when the optimum values of the step and the gamma are found, if the learning data is changed or the structure of the neural network is changed, there is a problem that the optimum values of the step and the gamma need to be found again.
In addition, in order to find the optimum values of the step and the gamma, it takes much time and effort to learn the neural network with the optimal values.